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(54) Client-server system for three-dimensional lace make-up simulation 



(57) It is an object of the invention to provide a three- 
dimensional beauty simulation client-server system 
which is capable of handling a user's face in a three- 
dimensional fashion and of providing more realistic 
beauty simulations. This system comprises a shop- 
based client that obtains and transmits three-dimen - 
sional shape data regarding a user, and a server com- 



prising a makeup simulation unit that receives and 
stores the three-dimensional shape data from the shop- 
based client and carries out makeup simulation based 
on the three-dimensional shape data in response to re- 
quests from the user, and a data control unit that ana- 
lyzes the user's operation record and generates admin- 
istrative information. 



F 1 G. 1 



CM 
< 

CO 



Q. 
UJ 



iTC. 



15 



IIMTEMCT CatULM TELFM0NE) 



CAUtt* 




Printed by Jouve. 75001 PARIS (FR) 



BEST AVAILABLE COPY 



1 



EP1 134 701 A2 



Description 

BACKGROUND OF THE INVENTION 

1. Field of the Invention s 

[0001 ] The present invention relates to a three-dimen- 
sional beauty simulation client-server system to carry 
out beauty simulations based on a user's face model 
data. 10 



2. Description of the Related Art 

[0002] The invention described in Japanese Patent 
Laid-Open No. H6-319613, for example, comprises a 
conventional beauty simulation apparatus. This inven- 
tion discloses a face makeup support apparatus using 
which makeup may be applied to a displayed face by 
simulating lipstick application, face powdering and eye* 
brow shaping on the image of a face displayed in an 
image display apparatus. 

[0003] The conventional beauty simulation apparatus 
entails the problem that it can only carry out flat image 
processing, and does not appear realistic. Furthermore, 
it cannot be used over a computer network. 

SUMMARY OF THE INVENTION 

[0004] The present invention was created in order to 
resolve these problems, and an object thereof is to pro- 
vide a three-dimensional beauty simulation client-server 
system that can display a user's face in a three-dimen- 
sional fashion and provide a more realistic beauty sim- 
ulation. 

[0005] The three-dimensional beauty simulation cli- 
ent-server system pertaining to the present invention in- 
cludes a shop-based client that obtains and transmits 
three-dimensional user data, a makeup simulation unit 
that receives and stores the three-dimensional shape 
data from the shop-based client and carries out makeup 
simulation based on the three-dimensional shape data 
in response to requests from the user, and a server that 
includes a data control unit that analyzes the user's op- 
eration record and generates administrative informa- 
tion. 

[0006] It is preferred that the shop-based client in- 
clude a plurality of cameras by which to obtain images 
of the user from a plurality of viewpoints, a correspond- 
ing point search unit that receives each item of image 
data obtained from the plurality of cameras, analyzes 
the plurality of images, and searches for corresponding 
points that match each other, a three-dimensional shape 
recognition unit that analyzes the searched correspond- 
ing points and recognizes the three-dimensional shape 
of the object, a geometric calculation unit that sets a line 
of sight based on the recognition results from the three- 
dimensional shape recognition unit and generates an 
image from the prescribed line of sight through geomet- 
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ric conversion of the data based on the set line of sight, 
a display unit that displays the image generated by the 
geometric calculation unit, and a communication means 
that transmits the image data generated by the geomet- 
ric calculation unit to the server. 
[0007] It is further preferred that the makeup simula- 
tion unit of the server include a receiving unit that re- 
ceives the three-dimensional shape data, a database 
that stores the received three-dimensional shape data, 
and a makeup simulation providing unit that provides a 
makeup simulation in response to requests for such sim- 
ulation, and that the data control unit of the server in- 
clude a user information analyzer that receives the op- 
eration history of the user and analyzes the trends there- 
in, a control database that stores the analyzed data, an 
information processing unit that reads out data from the 
control database in response to external requests and 
processes the data in accordance with the requests, and 
a transmitting/receiving unit that transmits the output of 
the information processing unit to the requesting source 
and receives requests from the requesting source. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] 

Fig. 1 is a drawing showing the overall system per- 
taining to an embodiment of the present invention; 

Fig. 2 is a drawing showing the basic construction 
of the server pertaining to the embodiment of the 
present invention; 

Fig. 3 is a drawing showing the basic construction 
of the shop-based client pertaining to the embodi- 
ment of the present invention; 

Fig. 4 is an example of the display screen of the 
shop-based client pertaining to the embodiment of 
the present invention; 

Fig. 5 is a flow chart showing the basic outline of 
the processing performed by the image processing 
apparatus pertaining to the embodiment of the 
present invention; 

Fig. 6 is a drawing to explain the operation principle 
cf the image processing apparatus pertaining to the 
embodiment of the present invention; 

Fig. 7 is a drawing to explain the operation principle 
of the image processing apparatus pertaining to the 
embodiment of the present invention; 

Fig. 8 is an external view of the image processing 
apparatus pertaining to the embodiment of the 
present invention; 
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Fig. 9 is a drawing to explain the operation principle 
of the image processing apparatus pertaining to the 
embodiment of the present invention; 

Fig. 1 0 is a drawing to explain the operation princi- s 
pie of the image processing apparatus pertaining to 
the embodiment of the present invention; 

Fig. 11 is a drawing to explain the operation princi- 
ple of another image processing apparatus pertain- io 
ing to the embodiment of the present invention; 

Fig. 12 is a summary block diagram of the image 
processing apparatus pertaining to the embodiment 
of the present invention; 15 

Fig. 13 is a flow chart showing the basic sequence 
to decide the camera orientations of the image 
processing apparatus pertaining to the embodiment 
of the present invention; 20 

Fig. 14 is a flow chart showing an outline^ of the 
match propagation sequence carried out by the im- 
age processing apparatus pertaining to the embod- 
iment of the present invention; and *5 

Fig. 15 is a drawing to explain the principle of mor- 
phing. 

DESCRIPTION OF THE PREFERRED 30 
EMBODIMENTS 

[0009] An embodiment of the present invention will be 
explained. - 

[0010] First, the concept of the present invention will 35 
be explained. 

[0011] When a consumer accesses an Internet-based 
'total beauty site' that is implemented by an embodiment 
of the present invention, data regarding the number of 
product click-throughs, etc. made by the consumer is 
obtained and analyzed, and a database is generated. 
[0012] 3D face model data (which includes not only 
the face but the overall body data) of the consumer is 
obtained by the apparatus located in the shop, and is 
stored in the server of the total beauty site*. A prescribed 
fee is paid by the shop to the operator of the site to cover 
such charges as a fee for use of the technology of the 
apparatus and the data, a franchise fee, the sales mar- 
gin, and a consulting fee. 

[0013] The site operator provides to manufacturers, 
magazine publishers and the like (1) consumer prefer- 
ence information derived from the number of click- 
throughs and (2) additional Information classified by 
age, etc. Conversely, manufacturers, magazine publish- 
ers and the like pay the site operator consulting fees and 
data fees. 

[0014] Next, a basic outline of the system pertaining 
to an embodiment of the present invention will be ex- 



plained with reference to Figs. 1 through 4. 
[0015] In Fig. 1, connected to the Internet 11 are a 
server 1 0 to operate the total beauty site', a shop-based 
client 12 to obtain 3D face model data, a shop-based 
client 13 to perform simulation, a client 14 by which a 
consumer can carry out a simulation at the consumer's 
home, and an Internet cellular telephone 15 having a 
camera 15a to obtain a 3D face model. A computer 16 
belonging to a manufacturer, magazine publisher, etc. 
may be connected to the server 10. Furthermore, the 
shop-based clients 12 and 13 may be connected to the 
server 10 through a method other than the Internet 11 . 
[0016] In Fig. 1, the consumer obtains her own 3D 
face model data in a beauty shop using the 3D image 
capturing apparatus 12a and the 3D image processing 
apparatus 12b connected to the shop-based client 12. 
The specific procedure followed will be discussed be- 
low. The shop-based client 12 sends the obtained 3D 
face model data to the server 10. The server 10 stores 
the received data. Once the data is stored in the server, 
the consumer can access the total beauty site 1 on the 
server 10 and carry out a makeup simulation from the 
shop-based client 12, the shop-based client 13 that has 
no image capturing apparatus, the home-based person- 
al computer 14 or the Internet cellular phone 15, The 
details of the makeup simulation will be described be- 
low. The behavior of the consumer while she is access- 
ing the total beauty site' is analyzed by the data control 
unit 10c and accumulated in a database. Because this 
data comprises information important in understanding 
the consumer's preferences, it is provided by the server 
10 to manufacturers, magazine publishers, etc. 16. 
Where the consumer has a camera-equipped computer 
or internet cellular phone, a plurality of images obtained 
therefrom may sent to the server, enabling the server 
10 to constructs 3D face model data. In the above ex- 
planation, the makeup simulation is carried out by the 
server 10. Because the simulation processing compris- 
es advanced processing including three-dimensional 
processing, by having it carried out by a server 10 ca- 
pable of advanced processing, the burden on the con- 
sumer-side personal computer may be reduced. It is ac- 
ceptable if the makeup simulation is carried out by the 
shop-based client 12. 

[0017] Fig. 2 shows the basic construction of the serv- 
er 10. The member registration unit 10a shown in Fig. 
1 includes the member registration unit 100 and mem- 
ber database 101 shown in Fig. 2. The makeup simula- 
tion unit 1 0b shown in Fig. 1 includes the 3D face model 
data receiving unit 102, the 3D face model database 103 
and the makeup simulation providing unit 104 shown in 
Fig. 2. The data control unit 1 0c shown in Fig. 1 includes 
the user information analyzer 105, the control database 
106, the information processing unit 107 and the trans- 
mitting/receiving unit 108 shown in Fig. 2. 
[001 8] When makeup simulation is carried out, mem- 
ber registration must first be performed. A member reg- 
istration request is sent by the shop-based client 12 or 
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the home-based computer 14 to the server 10. When 
the member registration request is received, the mem- 
ber registration unit 100 writes member information into 
the member database 101. 

[0019] The 3D model face data sent by the shop- 
based client 1 2 is received by the receiving unit 1 02 and 
stored in the 3D face model database 1 03. When a sim- 
ulation request is sent by the shop-based client 1 2 or 13 
or the home-based computer 14 to the server 10, the 
makeup simulation providing unit 104 determines 
whether or not the request is from a member, and if the 
request is from a member, it analyzes the contents of 
the request, reads out the member data from the data- 
base 103, carries out simulation in accordance with the 
request, and provides the simulation to the requesting 
party. 

[0020] At the same time, the actions taken by the con- 
sumer while at the total beauty site', for example, the 
contents of the simulation, clicks on specific products, 
clicks on banner ads, etc., are analyzed by the user in- 
formation analyzer 1 05 and are organized and stored in 
the control database 1 06 as consumer preference infor- 
mation. When an information supply request is received 
by the transmitting/receiving unit 108 from a manufac- 
turer, etc., the information processing unit 1 07 reads out 
prescribed data from the control database 1 06, subjects 
it to processing in accordance with the contents of the 
request, and then sends it to the requesting source. The 
details of the operation of the user information analyzer 
105 and the information processing unit 107 are ex- 
plained below. 

[0021] Fig. 3 shows the construction of the shop- 
based client 12. The shop-based client 12 includes a 
plurality of cameras 1a, 1b, etc., a 3D face model gen- 
erating unit 2, a makeup simulation unit 3, a display unit 
4, a touch panel 4a located on the display unit, a data- 
base 5 that stores 3D face model data, a pointing device 
6 such as a mouse, and a communication means 7 that 
connects to the server 10 or the Internet 1 1 . The 3D face 
model generating unit 2 includes a corresponding point 
search unit 2a, a three-dimensional shape recognition 
unit 2b and a geometric calcuiation unit 2c. Detailed ac- 
tions of these units will be described later. 
[0022] Fig. 4 shows the display apparatus 4. The 
three-dimensional image of the consumer is displayed 
in the area indicated by 4b, and color or pattern palette 
is shown in the touch panel 4a. Because a three-dimen- 
sional image is displayed in the display apparatus 4, a 
realistic makeup simulation may be experienced. For 
example, any type of makeup may be made through a 
one-touch operation of the touch panel 4a. A makeup 
style corresponding to various types of situations may 
be prepared in advance, such as party makeup, work 
makeup, etc., and the consumer's face may be repro- 
duced with the new makeup style based on a single 
touch of the touch panel. Alternatively, a makeup simu- 
lation may be executed manually. Once 3D face model 
data is sent to the server 10, these simulations may be 
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carried out on a personal computer at home. 

Makeup simulation 

5 [0023] The contents of a simulation carried out using 
the makeup simulation providing unit 1 04 and the make- 
up simulation unit 3 will now be explained. 
[0024] Using this simulation, simulations of makeup, 
cosmetic surgery, clothing, perfume, accessories, hair 
10 style, etc., may be provided based on 3D information. 
In addition, using morphing technology as described be- 
low, information to enable one to resemble one's favorite 
model may be obtained. For example, intermediate im- 
ages resembling a cross between oneseif and one's fa- 
's vorite model may be created through morphing technol- 
ogy, and the desired image may be selected. The user 
can learn what percentage of the image comprises her 
own features and what percentage comprises the mod- 
el's features. Simulation of not only one's face (the head 
20 area) but also one's entire body is possible as well. 
[0025] Several specific examples of such a simulation 
will now be explained. 

[0026] A simulation in which the level of beauty and 
degree of aging of the face are assessed and face iden- 

25 tification is carried out will first be explained. 

[0027] By analyzing the condition of ones facial skin 
and the light and dark areas reflecting the protrusions 
and indentations thereon, i.e., the state of the light areas 
and dark areas, the degree of one's beauty and appar- 

30 ent age can be objectively evaluated, identification of 
individual faces can be made, and the facial expression 
that best indicates and quantifies the person's emotional 
state can be objectively evaluated to a significant extent. 
[0028] Each individual's face has its own particular 

35 light and dark areas and areas comprising a mixture 
thereof. This means that each individual's face may be 
recognized based on these light, dark and mixed areas. 
Furthermore, depending on how one observes the face, 
one can observe changes in the shape of a person's 

<o light and dark areas that occur with changes in a per- 
son's facial expression, in other words, one's facial ex- 
pression changes with the contraction and relaxation of 
facial muscles, and such changes entail changes in the 
indentations and protrusions on one's face, i.e., in the 

*5 light and dark areas of the face. Therefore, by observing 
these changes, even such an imprecise concept as a 
person's 'expression' can be quantified and objectively 
evaluated. 

[0029] Accordingly, an evaluation facial image, which 
50 comprises a facial image that has undergone light/dark 
processing, is used. To create such an image, first, an 
image of the subjects face must be captured and a facial 
image obtained. Next, through image enhancement 
processing of this facial image, particularly Image en- 
55 hancement processing regarding the brightness of the 
image, an evaluation facial image comprising a plurality 
of areas having different levels of brightness is created. 
When this is done, the face may be evaluated for various 
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purposes, such as the degree of beauty or of aging in 
the face, based on the contours of the light and dark 
areas in the evaluation image, or on the borders be- 
tween these areas. 

[0030] Furthermore, by comparing the evaluation fa- 5 
cial image before and after face lift surgery, a determi- 
nation of the degree of aging may be carried out. 
[0031] Using the above processes, a simulation for 
plastic surgery or makeup styles may be performed. 
[0032] Next, the process by which to correct the facial io 
image will be explained. First, a desired face is chosen, 
images of a plurality of corrected candidate faces having 
varying degrees of resemblance to the desired face are 
created by alternating the original facial image using im- 
age processing such that the original facial image re- is 
sembles the desired face image, and a corrected facial 
image is obtained by selecting from among these a plu- 
rality of corrected candidate facial images. 
[0033] A model face may be used to choose the de- 
sired face. For the model face, a favorite television per- 20 
sonality or actress may be used. 
[0034] First, the desired face is selected. Where a 
makeup instructor is providing guidance regarding 
makeup application to a person wishing to wear make- 
up, for example, the desired face is chosen through the 23 
makeup instructor asking the prospective makeup wear- 
er about her preferences. The desired face may be cho- 
sen using a model face. For the model face, the pro- 
spective makeup wearer can use the face of a favorite 
television personality or actress. 30 
[0035] When the desired face is chosen , virtual make- 
up faces based on the desired face, i.e., images of vir- 
tual faces having the desired makeup style, are created. 
Through image processing such as the fusing of the de- 
sired face with the image of the face of the prospective 35 
makeup wearer, the faces of the prospective makeup 
wearer and of the desired face may be combined, bring- 
ing the prospective makeup wearer's face closer in ap- 
pearance to that of the desired face. From these virtual 
makeup faces, the ideal makeup face that is most de- <o 
sired by the prospective makeup wearer is determined. 
Specifically, because images of a plurality of virtual 
makeup faces exhibiting varying degrees of fusion or re- 
semblance between the prospective makeup wearer's 
face and the desired face are obtained through the 
above image processing, the preferred face can be cho- 
sen from among these faces as a desired virtual makeup 
face within the range of resemblance levels that may be 
obtained through the application of makeup. In this way, 
the ideal makeup face that is anticipated to be ultimately so 
obtained may be provided beforehand. In other words, 
the prospective makeup wearer can learn the final 
made-up look in a short amount of time. By selecting the 
desired face forming the basis for the makeup and seek- 
ing the ideal makeup face to obtain in connection with ss 
this face in this way, the final made-up look may be dis- 
played in a short amount of time. 
[0036] Once the ideal makeup face is chosen through 



the above makeup simulation process, a makeup tech- 
nique is deduced from the ideal makeup face. In other 
words, a series of makeup pointers by which to obtain 
the desired look, such as the areas where the eyebrows 
should be plucked or darkened, lines and areas where 
eye liner and eye shadow should be applied, eye shad- 
ow colors, areas where lipstick should be applied, and 
techniques for the application of foundation, are deter- 
mined based on a preset makeup program. Makeup is 
then applied to the prospective makeup wearer's face 
based on these makeup pointers. As a result, the ideal 
makeup face, i.e., the look that was accepted before- 
hand by the prospective makeup wearer, may be accu- 
rately reproduced on the face of the prospective makeup 
wearer. Put another way, any makeup desired by the 
prospective makeup wearer can be applied on her face 
in a short period of time. 

[0037] As described above, the makeup method of 
the present invention is characterized in that an ideal 
face based on a desired face, i.e., a model face, is cre- 
ated through image processing, and an important as- 
pect of this method is that the current face of the pro- 
spective makeup wearer and the model face are incre- 
mentally combined and brought closer together through 
image processing. 

[0038] Through this process, advanced corrections to 
the facial image may be made easily and in a short pe- 
riod of time. Furthermore, any desired makeup may be 
applied on the prospective makeup wearer in a short 
amount of time, and makeup possibilities based on a 
wide variety of cosmetic products may be effectively uti- 
lized. 

[0039] Next, the simulation of makeup facial images 
in beauty parlors, cosmetics shops, beauty schools, etc. 
will be explained. 

[0040] Makeup simulation drawing software uses a 
method in which the face as a whole is made up by ap- 
plying makeup to individual parts of the face. In this 
method, the sought makeup style is pasted on. For ex- 
ample, regarding eyebrows, a method is used in which 
a given eyebrow shape is chosen and pasted onto the 
existing eyebrow after it is matched to the size of the 
eyebrow on the facial image. Similarly, where lipstick is 
applied to the lips, a method is used in which a pre-ex- 
isting form is pasted on to an image. 
[0041] For the model makeup, images of the eye- 
brows, lipstick on the lips, powder (including foundation, 
eye shadow and blush) on the skin, and colored contact 
lenses are drawn. The image drawing operation for each 
facial component is explained below. 

• Eyebrows: The eyebrow area is defined, the eye- 
brow in the original eyebrow area is shaved off, and 
the color of the surrounding skin is drawn in. An eye- 
brow shape is chosen, and that shape is drawn in 
the eyebrow area. When this is done, processing is 
performed on a pixel-by-pixel basis in the eyebrow 
area, and the eyebrow is drawn in accordance with 
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a defined calculation formula. 

• Lipstick: The lip area onto which lipstick is to be ap- 
plied is defined, and the chosen lipstick color is ap- 
plied to the lip area. Here, image processing is car- 5 
ried out using the three elements of the hue, bright- 
ness and saturation. An image of the lip area is 
drawn by replacing the lip with the hue of the lipstick 
color, and converting the brightness and saturation 

of the original lips to the brightness and saturation u 
of the lipstick. When this is done, operations such 
as glossing are also performed. Furthermore, the 
areas around the border between the lips and the 
skin are drawn such that the border between the 
lips and skin is a continuous color. is 

• Powdering of the skin: An image is drawn in which 
the skin color value and the powder color value are 
mixed according to a specified ratio. Here, powder- 
ing includes the application of makeup such as 20 
foundation, eye shadow and blush. 

• Colored contact lenses: After the positions at which 
colored contact lenses are to be placed in the image 
(the positions at which the colored parts of the con- 25 
tacts are to be drawn) are defined, the color values 

of the colored contact lenses and of the iris are 
mixed according to a defined ratio in the display. 

[0042] The following variables should be recorded as 30 
makeup information for the model on whom makeup is 
applied using the above methods: 

• Eyebrows - eyebrow shape, eyebrow color (color 
value), position and size relative to the face 35 

• Lipstick - lipstick color value, degree of gloss 

• Powder - powder color value and density of appli- 
cation on a pixel-by-pixel basis ao 

• Colored contact lenses - color value of colored con- 
tact lenses 

• Facial image definition points 45 

[0043] The processing to apply the makeup of the se- 
lected model to the facial image of the user is carried 
out according to the following procedure. First, in a pre- 
liminary step, the facial image of the user is loaded into so 
a computer using a digital camera, etc., and the -facial 
image is defined to match the image set for the model. 
Afterward, the same attribute values used for the mod- 
el's makeup are loaded in and applied to the defined 
facial image of the user. While the makeup is different 55 
for each facial component, the same materials are used 
for the user's eyebrows, lipstick and colored contact 
lenses that were used for the model, and images are 



drawn in the user's facial image using the same methods 
that were used for the drawing of the model face. How- 
ever, regarding powdering, because the density and the 
type of powder may differ depending on the area of the 
face, the correspondences of the respective pixels of the 
facial images of the model and the user are obtained, 
and makeup is applied to the facial image of the user. 
Specifically, using morphing technology, the corre- 
spondence of the respective pixels of the model facial 
' image and the user's facial image is calculated, and the 
same powder that was present in a given pixel of the 
model image is used for the pixel of the user's skin hav- 
ing the same skin attribute at the same pixel of the model 
image. 

[0044] When a desired model facial image is selected 
from among the plurality of model facial images dis- 
played on the screen (where the makeup varies even 
though the same model is used, the different images are 
displayed as a separate menu item), the eyebrows, lip- 
stick, and colored contact lenses are automatically ap- 
plied using the methods explained with regard to the 
model makeup. However, because the powder applica- 
tion can differ on a pixel-by-pixel basis, powder is ap- 
plied using morphing technology. Two techniques are 
used in morphing; warp and dissolve. The first involves 
a method by which, when changing from one shape to 
another, the corresponding points are sought and the 
original shape is transformed, while the second involves 
transformation of the image by mixing the pre-change 
color and the post-change color in accordance with a 
defined ratio. The image drawing carried out for pow- 
dering in the present invention uses warp. 
[0045] It is often difficult to determine precisely what 
type of makeup one desires. Therefore, it is useful to 
have a model makeup style to refer to. When several 
models are registered beforehand, the makeup style 
used by the model can be applied to the user's facial 
image simply through the selection of a particular model, 
and therefore makeup styles that the user likes, or mod- 
el makeup styles that might be applied to one's face, 
can be selected, and many different makeup styles may 
be 'tried on'. 

[0046] Not only can the makeup style that was applied 
to the model facial image be transferred to the user's 
facial image, but because makeup is applied while pre- 
serving the user's facial contours and skin features, dif- 
ferent impressions may be created even with the same 
makeup style. For example, powder applied to a paie- 
skinned model reflects light differently than powder ap- 
plied to a user's suntanned skin. Consequently, the best 
makeup style for one's own face can be sought through 
simulation on a screen. 

[0047] Once the user's facial image data is loaded into 
a computer via an image data input apparatus and the 
facial area definition is made, simulation may be per- 
formed by changing the model an unlimited number of 
times. Furthermore, the skin color, the condition of the 
lips, or the quality of the face itself can change between 
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a facial image taken in summer and a facial image taken 
in winter. In the present invention, because makeup is 
applied while the characteristics of the original facial im- 
age are preserved, and these characteristics may 
change from time to time as described above, the effect 5 
of the makeup at different times may be checked and 
confirmed even if the same makeup is applied (i.e., the 
same model is selected) to the face of the same person. 
For example, because the method of the present inven- 
tion by which lipstick is drawn on the lips preserves the to 
lines and shading of the lips even after the application 
of lipstick, one can clearly see the difference in the effect 
of the lipstick between when it is applied on rough lips 
during winter and when it is applied on fresh lips during 
summer. In the case of skin in particular, because skin « 
color differs in summer and winter, the effect of makeup 
varies depending on the type of powder used. Using the 
automatic makeup simulation of the present invention, 
the differences in the effect of makeup based on the con- 
dition of the user's face may be directly confirmed on a 
screen as described above. Therefore, if facial images 
taken in the four different seasons are used, the best 
makeup style may be found in a short amount of time 
by applying the makeup styles of various different mod- 
els on one's facial image for the current season. « 

User information analyzer 

[0048] Next, the user information analyzer 105 and 
the user processing unit 1 07 will be explained. These 30 
units carry out the following processes: 

(1) Statistical compilation of the user's Web usage 
information (number of click-throughs), etc.) 

(2) Analysis of the Web usage information, analysis 
of not just purchasing information but preference in- 
formation 

(3) Aggregation of purchasing information, supply 
of product preference information in new form 

(4) Analysis cross-referenced by age and region in- 
formation 

[0049] The user information analyzer 1 05 extracts, or- 
ganizes and supplies data by which to understand the 
overall user information based on the contents of the 
member database 101. It performs classification of all 
of the registered users, and outputs basic user charac- 
teristic data such as the total number of registered us- 
ers, the ratio of men to women, the distribution of users 
by age and location of residence, etc. From the user be- 
havior history, which includes information on the degree 
of cooperation of each user with questionnaire surveys 
and on the frequency with which the user purchased 
products through the Internet home page, the class of 
users that would be best selected as target users may 
be learned. 

[0050] If the attributes of the target user class are 
clearly established, an effective business may be devel- 
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oped by matching with the preferences of the target user 
class such basic elements as the contents of the total 
beauty site', the style of writing, and the merchandise 
offered. Furthermore, problems may become more 
clearly defined. An example of such a problem might be 
that although women were originally targeted, there are 
fewer female registered users than expected. In such a 
case, responsive actions such as the posting of banner 
ads in information portals or sites accessed by a large 
number of women may be taken. 
[0051] It is also possible to prepare a number of dif- 
ferent brochures that are custom -tailored to the at- 
tributes of each group of users, and to send brochures 
with different contents to each group. One such bro- 
chure might focus on information regarding the most 
popular products among the merchandise handled by 
the total beauty site'. A better response would be antic- 
ipated in such a case that would be expected when the 
same brochure with the same content is sent to all mem- 
bers on a global basis. 

[0052] The user information analyzer 105 performs 
access analysis. 'Access analysis' is the most basic 
analysis that measures the number of people that visit 
a particular site. If a site is equated to a shop, this 
number is eq uivalent to the number of customers visiting 
the shop. Analysis from various viewpoints may be car- 
ried out. For example, trends may be obtained regarding 
the number of customers visiting on each day of the 
week or during each time period of the day, the number 
of customers who enter but leave without purchasing, 
and the number of customers visiting each area of the 
site. 

[0053] Access analysis is performed using the three 
indices of number of hits, PV (page view), and number 
of visitors. 

[0054] The number of hits is a value that indicates the 
number of 'data sets* that were requested to be sent 
from a particular site. The unit of measurement for 'data 
sets' here is the number of data files in a computer. If 
the data set is a home page and the home page includes 
a large amount of graphic data, the number of hits in- 
creases accordingly. Conversely, even if a large amount 
of information is contained in one page, if that data con- 
sists of one text file, it is counted as T hit. 
[0055] A more practical index is PV (page view). It in- 
dicates the total number of Internet home pages viewed 
in connection with a particular site. While this index en- 
tails the shortcoming that any single home page counts 
as 1 PV regardless of the amount of information con- 
tained therein, it is a standard index used to measure 
the value of a medium or the effect of an ad, such as a 
banner ad, that is displayed on a one-page basis. 
[0056] There are cases in which the number of PVs 
associated with the top page of a particular site is 
deemed the number of visitors. Because PV indicates 
the number of total viewed pages, the number of differ- 
ent people that have viewed the page cannot be ob- 
tained. The number of visitors is an index for solving this 
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problem. Naturally, where one person accesses the top 
page repeatedly, each access is counted, and therefore, 
the number of visitors in this case is only an approximate 
number. 

[0057] In order to measure the number of visitors 
more precisely, such methods as a 'cookie' or a 'regis- 
tration system' must be used. 
[0058] A cookie not only enables behavior analysis, 
but is also effective for one-to-one marketing. The use 
of a cookie allows the behavior of a particular person (or 
more accurately, the behavior of a Web browser) within 
the site to be tracked. 

[0059] For example, suppose it is learned that con- 
sumers who request a lipstick simulation during a make- 
up simulation session are significantly more likely to re- 
quest lipstick brochures than consumers who do not re- 
quest a lipstick simulation. 

[0060] If this trend is utilized property, the target pop- 
ulation may be approached more effectively If a bro- 
chure request page is forcibly shown to users who re- 
quest a lipstick simulation, the rate of brochure requests 
may be increased substantially. 
[0061] Through the use of a cookie, information may 
be provided in a customized fashion that matches each 
user's behavior and preferences. In order to implement 
this feature, the site must have cookie issuance and da- 
tabase functions. 

[0062] While personalization based on the use of a 
cookie cannot completely specify each individual, a reg- 
istration system can overcome this shortcoming. 
[0063] The address, telephone number, e-mail ad- 
dress and name are registered beforehand, and an ID 
and password used exclusively by the lota! beauty site' 
are issued. A member accessing a site enters a mem- 
ber-only page when she inputs her ID and password. 
[0064] By having the users log in , the identity of each 
user, the pages they visit, and their behavior while 
logged in can be tracked by the site. At the same time, 
a page dedicated to the user may be displayed after 
login. 

[0065] If the areas of information desired by a user 
are obtained through responses to a questionnaire dis- 
tributed at the time of registration, news that matches 
the user's stated interests may be posted on a particular 
page. 

[0066] From not only the registration information, but 
also from behavior information that indicates the areas 
of the site most commonly visited by the user, the indi- 
vidual's preferences may be derived and information 
matching these preferences may be displayed. 

Three-dimensional face model generating unit 

[0067] The three-dimensional face model generating 
unit 2 will now be explained. 

[0068] One known method of image processing is the 
morphing technique. Morphing is a computer graphics 
(CG) technology developed in Hollywood, U.S.A. Ac- 



cording to this method, two different images are used, 
for example, images of the faces of two persons, and 
one of the images is gradually changed on the screen 
to the other image, thereby providing a series of images 
s showing such change. Using the morphing technology, 
it is possible to create a series of images in which, for 
example, a white tiger turns into a young woman. 
[0069] When two images A and B are given, the mor- 
phing process is roughly as follows. First, the corre- 
w sponding feature points between image A and image B 
are obtained (e.g., eye and eye, nose and nose). This 
process is normally performed by an operator. When the 
correspondences are found, feature point p of image A 
is gradually changed in a time-consuming process to 
is feature point q of image B, resulting in the image series 
as described above, 

[0070] In CG, an image is generally made of a large 
number of triangular elements. Therefore, morphing is 
performed by changing the triangle of feature point p in 
image A to the triangle of feature point q in image B while 
maintaining the correspondence between them. This 
will be described further with reference to Fig. 1 5. In this 
figure, triangle A is part of image A, and triangle B is part 
of image B. The apexes p1 , p2, p3 of triangle A each 
correspond to apexes q1 , q2 and q3 of triangle B. In or- 
der to convert triangle A to triangle B, the differences 
between p1 and q1 , p2 and q2, and p3 and q3 are cal- 
culated, and then respectively added to each of the 
apexes p1 , p2, p3 of triangle A. By adding all (100%) of 
these differences, triangle A is converted to triangle B. 
It is also possible to add portions of these differences 
instead of the whole differences, e.g., 30% or 60% 
thereof, in such case, the intermediate figures between 
triangle A and triangle B can be obtained. For example, 
in Fig. 15, triangle A* is a model example of an addition 
of 30% of the difference, and triangle B' is a model ex- 
ample of an addition of 60% of the difference. 
[0071] Fig. 5 is a flowchart showing an outline of the 
processing of the apparatus/method according to an 
embodiment of the present invention. The image data 
(signals) obtained from a plurality of cameras 1a, 1b, in 
Fig. 3. are input into a front view image generating unit 
2. In front view image generating unit 2, a corresponding 
point searching unit 2a searches the mutually corre- 
sponding points by analyzing the plurality of images. 
These corresponding points are analyzed by a three- 
dimensional shape identifying unit 2b, and the three-di- 
mensional shape of the object is identified. Based on 
the identified results, the viewing rays are set, and the 
data is geometrically converted or varied based on the 
set viewing rays, thereby generating a front view image 
that would be gained by looking into a mirror. Each of 
the above-mentioned processes will be described In fur- 
ther detail below. Furthermore, camera 1 need only be 
a plurality of cameras, regardless of whether 2, 3, 4 or 
more. Two or three are desirable from the practical as- 
pect. 

[0072] The processing of the front view image gener- 
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ating unit will be described in further detail based on 
Figs. 6 and 1 0. Fig. 6 is a model view of a digital mirror 
comprising cameras 1 at the left and right upper ends 
and the lower center of a plate-shaped liquid crystal dis- 
play apparatus (LCD) 4. An object 1 00 is placed on the s 
normal vector intersecting substantially the center of 
LCD 4. Normally, the face of the user is located at this 
position, but for convenience of explanation, a quadran- 
gular pyramid is used as an example. When quadran- 
gular pyramid 100 is shot by cameras 1a, 1b, and 1c, 10 
images 100a, 100b, and 100c are obtained. Image 100a 
is shot by camera la, and viewed from LCD 4, this image 
is a view of pyramid 100 from the left side. Image 100b 
is shot by camera 1 b, and is a view of pyramid 1 00 from 
the right side. Image 100c is shot by camera 1c, and is 
a view of pyramid 100 from the bottom. If there are at 
least two images seen from different viewpoints located 
relatively adjacent to each other, then it is possible to 
identify a unique three-dimensional shape from a plural- 
ity of two-dimensional images through a geometrical & 
calculation processing similar to the stereoscopic view 
processing. In order to perform this processing by a 
computer, it is necessary to specify the feature points. 
In the present example, the apexes of quadrangular pyr- 
amid 100 are selected. When the feature points have 23 
been specified for all images, the correspondence be- 
tween these feature points is calculated. In this way, it 
is analyzed at which position in each image the same 
portion of pyramid 100 is located. Based on this analy- 
sis, the three-dimensional shape of pyramid 1 00 is iden- 
tified. According to image 100a, the apex is on the left 
side, so it is clear that pyramid 1 00 is at the left of cam- 
eral la. in this way, the three-dimensional shape is iden- 
tified. Thereafter, the viewpoint is set for example sub- 
stantially in the center of LCD 4, and based on this view- 
point, an image of pyramid 1 00 is generated. For exam- 
ple, image 100 as shown in Fig. 7 is obtained. 
[0073] In Fig. 3, signal processing unit 3 receives the 
front view image processed as above from front view 
image generating unit 2, and performs various process- 
ing such as displaying the object or a reflection of the 
object such as gained by conventional mirror reflection, 
etc. Examples are the zoom and wide angle processes. 
A certain portion of the whole image reflected in a mirror 
is instantaneously enlarged or reduced. The selection 
of the portion to be enlarged or reduced and the 
processing to be performed is designated by a pointing 
device 6 such as a mouse, tf the surface of LCD 4 is a 
touch panel, it is possible to touch an arbitrary portion 
of the image to enlarge or reduce such portion instan- 
taneously. 

[0074] Fig. 8 is a variation of the apparatus in Fig. 3. 
Three CCD cameras 1 a, 1 b and 1 c are provided around 
LCD 4. At the back of LCD 4, a computer is provided 
which functions as front view image generating unit 2 
and signal processing unit 3. These are ail stored in one 
case. 

[0075] Now, the whole processing of the apparatus/ 



method according to an embodiment of the present in- 
vention will be described in outline. According to the 
flowchart in Fig. 5, two or more images A, B,... from two 
or more different viewpoints are obtained (S1). 
[0076] Next, the correspondence between feature 
points in image A and image B is calculated (S2). Fea- 
ture points may be edges, comers, texture, etc. 
[0077] The difference between corresponding feature 
points in image A and image B is calculated (S3). 
Through this processing, the extraction of the necessary 
features points and the difference between them 
(amount of change) can be gained as required for the 
morphing process. 

[0078] The present embodiment is also a drawing ap- 
paratus and method for performing morphing of images 
of three-dimensional objects. In order to draw images of 
three-dimensional objects, the position of the object 
within a space must be determined, and, according to 
the present drawing apparatus and method, it is possi- 
ble to draw images of three-dimensional objects without 
directly requiring the three-dimensional position. 
[0079] The movement principle will be described by 
using Figs. 9 and 10. As shown in Figs. 9(a) and (b), a 
cone 201 and a cube 202 are arranged within a certain 
space and shot by two cameras la and 1 b. As the view- 
points of cameras 1 a, 1 b differ, the obtained images are 
also different. The images obtained by cameras la, 1b 
are as shown in Figs. 10(a) and (b). Comparing these 
two images, it is clear that the positions of cone 201 and 
cube 202 are different. Assuming that the amount of 
change in the relative position of cone 201 is y, and that 
of cube 202 is x, then Fig. 1 0 shows that x<y. This is due 
to the distance between the object and the cameras. If 
the values of x and y are large, the feature points are 
near the camera. On the other hand, if such values are 
small, the feature points are far from the camera. In this 
way, the distances between the object and the cameras 
are clear from the differences between corresponding 
feature points in the different images. Utilizing this char- 
acteristic, the feature points are sorted according to the 
differences (S4), and the images are written in order 
from that with the smallest difference (meaning the im- 
age shot by the camera farthest to the object) to the larg- 
est difference (S5). Portions near the camera are over- 
written and displayed, but portions far from the camera 
(hidden portions) are deleted through the overwriting. In 
this way, it is possible to adequately reproduce an image 
in three-dimensional space without using depth informa- 
tion. 

[0080] The apparatus shown in Figs. 5 to 8 is able to 
display an image shown from a different viewpoint than 
camera 1 by processing the image obtained from cam- 
era 1 as shown in Fig. 5. For example, It is possible to 
use the images of a face from the right, from the left and 
the bottom to generate and display the image of the face 
seen from the front. Also, by applying morphing 
processing to the face seen from the right and the left, 
it is possible to display the face from various angles, as 
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if the camera viewpoint had continuously moved. The 
apparatus in Figs. 5 to 8 can be used quasi as a digital 
form mirror (hereinafter the 'digital mirror"). 
[0081 ] It is also possible to use the apparatus in Figs. 
3 to 5 as a digital window simulating an actual window. 
By displaying various scenes on a liquid crystal televi- 
sion, the present invention provides a display apparatus 
for the window to be used in substitution of the actual 
window. Conventional display apparatuses merely dis- 
played images, e.g. scenery, seen from a fixed view- 
point, being unable to express small changes in scenery 
occurring from changes in the viewpoint position at the 
actual window. By utilizing the apparatus or method ac- 
cording to the present embodiment, ft is possible to rec- 
ognize the position of the person, i.e. the position of the 
viewpoint, so by changing the display according to the 
viewpoint position, an even more real scenery display 
is possible. For example, Fig. 11 shows a liquid crystal 
apparatus ("digital window") W and a person standing 
before it. In Fig. 11 (a), a cone and cube are arranged 
within a virtual space, and this situation is displayed on 
liquid crystal apparatus W. If the person is at position b, 
the image shown in Fig. 11 (b) will be displayed on liquid 
crystal apparatus w, and if the person is at position c, 
then the image shown in Fig. 11(c) will be displayed. In 
this way, by displaying an adequate screen according 
to the viewpoint position, the user will feel as if he were 
turning his head at an actual window. 
[0082] The digital mirror and digital window process- 
ing methods above are common in that they include a 
processing of determining the position of an object with- 
in a three-dimensional space by calculating the corre- 
spondence of feature points between a plurality of im- 
ages. In the digital mirror, the position measurement 
precision is desirably high, as the measurement preci- 
sion of the three-dimensional position directly affects the 
image precision. However, in the digital window, there 
is no large feeling of strangeness even if the viewpoint 
position is somewhat inaccurate. Therefore, the digital 
window does not require as high a measurement preci- 
sion of the position as the digital mirror. A processing 
apparatus/method for the digital mirror will be hereinaf- 
ter referred to as the facial image generator and a 
processing apparatus/method for the digital window the 
scenery image generator. Both will be now described in 
further detail. 

[0083] The facial image generator conducts its 
processing using three cameras and a trifocal tensor 
suited as constraint. The scenery generator conducts 
its processing using two cameras and the epipolar ge- 
ometry as constraint. Conventionally, it was difficult to 
find correspondences only by comparing the three im- 
ages of the three cameras, but by using the space con- 
straints of the three cameras, the correspondence 
search can be performed automatically. 



Facial Image Generator 

[0084] An example of the processing of three images 
with different viewpoints from three cameras will be de- 
5 scribed below. 

1. Feature point detection unit 

[0085] Three images with different viewpoints are in- 
io put into three feature point detection units 10a to 10c. 
Feature point detection units 10a to 10c outputs a list of 
feature points also called points of interest. If the object 
has a geometrical shape such as triangles or squares, 
the apexes thereof are the features points. In normal 
'5 photograph images, points of interest are naturally good 
candidates for feature points as points of interest are by 
their very definition image points that have the highest 
textureness. 



[0086] Correlation units Ha and lib and a robust match- 
ing unit 12 make a seed finding unit. This unit functions 
to find an aggregate of initial trinocular matches (con- 
straint of the positions of three cameras) that are highly 
reliable. Three lists of points of interest are input into this 
unit, and the unit outputs a list of trinocular matches of 
the points of interest called seed matches. Correlation 
units Ha and 11b establish a list of tentative trinocular 
matches. Robust matching unit finalizes a list of reliable 
seed matches using robust methods applied to three 
view geometric constraints. 

2.1 Correlation unit 

The movements of correlation units Ha and 11b 
will be described below. These units perform the 
processing of three lists of points of interest in three 
images output from feature point detection unit 1 0a 
to 1 0c. The ZNCC (zero-mean normalized cross- 
correlation) correlation measure is used for finding 
correspondences. By using the ZNCC correlation 
measure, it is possible to find the correspondence 
between images even if the size of the object is 
somewhat different between such images or the im- 
ages are somewhat deformed. Therefore, the 
ZNCC correlation is used for matching seeds. 

The ZNCCx(A) at point x = (x,y) T with the shift 
A = (A x , A Y ) T is defined to be: 

£,(/(x + i) - 7(x))(/'(x + A + 1) - 7'(x + A)) 
(I,(/(x + i) - 7(x)) 2 E,(/'(x + A + i) - 7'(x + A)) 2 ) 1/2 

where l*(x) and H(x) are the means of pixel lumi- 
nances for the given window centered at x. 

2.2 Robust matching unit 

Next, the binocular matches from correlation 
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unit 11 are merged into one single trinocular match 
by robust matching unit 12. Robust matching unit 
12 receives input of a list of potential trinocular 
matches from correlation unit 1 1 and outputs a list 
of highly reliable seed trinocular matches. A robust 
statistics method based on random sampling of 4 
trinocular matches in three images is used to esti- 
mate the 12 components of the three-view con- 
straints to remove the outliers of trinocular matches. 
When the same object is shot by three cameras and 
three images from different viewpoints are gained, 
the same point in the object in each of the three im- 
ages (e.g., position of feature point) can be uniquely 
defined from the position of the object, the camera 
position and the camera direction according to cer- 
tain rules. Therefore, by determining whether the 
points of interest in the list of trinocular matches 
gained from correlation unit 11 satisfies such rules, 
it is possible to obtain the list of points of interest of 
the correct trinocular matches. 

Given u = (u,v), u' = (u'Y) and u" = (u B ,v") the 
normalized relative coordinated of the trinocular 
matches, the three-view constraints are completely 
determined by the following 1 2 components t, to t 12 : 

t A u+ <a v+f n i/ + tg u" = 0, 



fc> u+f 6 v+f 11 v , + f 10 t/" = 0, 
f 3 u + f 7 v + f 12 t/ + f g ✓* = 0, 



v" = 0. 



3 Unit of auto-determination of camera orientations 

[0087] Now, a camera orientation auto-determination 
unit 1 3 will be described below. The classical off-line cal- 
ibration of the whole system is hardly applicable here 
even though 3 cameras may be a priori fixed, but their 
orientations could be still variable. Therefore, camera 
orientation auto-determination unit 13 determines the 
camera orientation in order to constrain the match prop- 
agation. In other words, camera orientation auto-deter- 
mination unit 13 receives input of a list of seed matches 
from robust matching unit 12 and outputs the orientation 
of the camera system. 

[0088J Now, the basic ideas of camera orientation au- 
to-determination unit 1 3 will be described below. At first, 
the three-view constra ints tj , .... t, 2 are optim ally re-com- 
puted by using all trinocular inlier matches. The extrac- 
tion of camera orientations directly from the three-view 
constraints for later usage is based on an original ob- 
servation that the problem of affine cameras is convert- 
ed into a nice problem of 10 projective cameras. 



[0089] For those skilled in the art, it is evident that an 
elegant 1D projective camera model first introduced in 
L. Quan and T. Kanade 'Affine structure from line cor- 
respondences with uncalibrated affine cameras" IEEE 
5 Transactions on Pattern Analysis and Machine Intelli- 
gence, 1 9(8): 834-845, August 1 997 occurs on the plane 
at infinity for the usual affine cameras. All directional 
quantities are embedded on the plane at infinity, there- 
fore encoded by the 1 D projective camera. The 1 D cam- 
io era is entirely governed by its trifocal tensor T 1J|C (provid- 
ing a strong constraint) such that T^ukj'Ju^ = 0. 
[0090] From the above aspects, the procedure of de- 
termining the camera orientations according to the 
present embodiment is as follows. 

15 

S11 : Convert 2D affine cameras into 1D projective 
cameras 

Using tensor-vector mapping defined by 4(a - 1) + 
2(b - 1 ) + c 1 between the tensor components and 
& the three-view constraint components converts the 
triplet of affine cameras represented by ti into the 
triplet of 1D cameras represented by T^. 
S12: Extraction of epipoles 
The 1 D camera epipoles can be extracted from the 
25 tensor by solving, for instance, IT. Jk e 2 l = 0 for. the 
epipoles % an d ©3 » n the first image. The other 
epipoles can be similarly obtained by factorizing the 
matrix T^e 1 , for e' t and e' 3 and T.^e^ for e% and 
e V 

30 S1 3: Determination of camera matrices M* = (H, h) 
and M" = (H\ h') and the camera centers c, c' and c" 
It is first straightforward that h = a', and h' = e", . The 
homographic parts of the camera matrices are de- 
termined from T^ = H,lhk - MH',*. Then, the camera 
centers and the 20 projective reconstruction can be 
determined from the camera matrices as their ker- 
nels. 

S14: Update of the projective structure 

The known aspect ratio for the affine camera is 

*o equivalent to the knowledge of the circular points 
on the affine image plane. The dual of the absolute 
conic on the plane at infinity could be determined 
by observing that the viewing rays of the circular 
points of each affine image plane are tangent to the 

45 absolute conic through the camera center. 

S15: Determination of camera orientation parame- 
ters 

Transforming the absolute conic to its canonical po- 
sition therefore converts all projective quantities in- 
50 to their true Euclidean counterparts. Euclidean 
camera centers give the orientation of the affine 
cameras and the affine epipolar geometry is de- 
duced from the epipoles. 

55 4. Constraint match propagation unit 

[0091] Now, a constraint match propagation unit 14 
for expecting a maximum number of matches in three 
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images will be described below. This unit 14 receives 
input of a list of seed matches and camera orientation 
parameters from camera orientation auto-determination 
unit 13 and outputs dense matching in three images. 
[0092] After obtaining the initial seed matches, it 
comes the central idea of match propagation from the 
initial seed matches. The idea is similar to the classic 
region growing method for image segmentation based 
on the pixel homogeneity. The present embodiment 
adopts region growing to match growing. Instead of us- 
ing the homogeneity property, a similarity measure 
based on the correlation score is used. This propagation 
strategy could also be justified as the seed matches are 
the points of interest that are the local maxima of the 
text uren ess, so the matches could be extended to its 
neighbors which have still strong textureness though not 
a local maxima. 

[0093] All initial seed matches are starting points of 
concurrent propagations. At each step, a match (a, A) 
with the best ZNCC score is removed from the current 
set of seed matches (S21 in Fig. 1 4). Then new matches 
are searched in its "match neighborhood' and all new 
matches are simultaneously added to the current set of 
seeds and to the set of accepted matehes-under con- 
struction (S22). The neighbors pixels a and A are taken 
to be all pixels within the 5 x window centered at a and 
A to ensure the continuity constraint of the matching re- 
sults. For each neighboring pixel in the first image, we 
construct a list of tentative match candidates consisting 
of all pixels of a 3 x 3 window in the neighborhood of its 
corresponding location in the second image. Thus the 
displacement gradient limit should not exceed 1 pixel. 
This propagation procedure is carried out simultaneous- 
ly from the first to the second and the first to the third 
image, and the propagation is constrained by the cam- 
era orientation between each pair of images. Only those 
that satisfy the geometric constraints of the camera sys- 
tem are propagated. Further, these two concurrent prop- 
agations are constrained by the three-view geometry of 
the camera system. Only those that satisfy the three- 
view geometry of the camera system are retained. 
[0094] The unicity constraint of the matching and the 
termination of the process are guaranteed by choosing 
only new matches not yet accepted Since the search 
space is reduced for each pixel, small 5x5 windows are 
used for ZNCC, therefore minor geometric changes are 
allowed. 

[0095] It can be noticed that the risk of bad propaga- 
tion is greatly diminished by the best first strategy over 
all matched seed points. Although seed selection step 
seems very similar to many existing methods for match- 
ing points of interest using correlation, the crucial differ- 
ence is that propagation needs only to take the most 
reliable ones rather than taking a maximum of them. 
This makes our algorithm much (ess vulnerable to the 
presence of bad seeds in the initial matches. In some 
extreme cases, only one good match of points of interest 
is sufficient to provoke an avalanche of the whole tex- 



tured images. 

5. Re-sampling unit 

5 [0096] Now, a re-sampling unit 15 will be described 
below. The dense matching may still be corrupted and 
irregular, re-sampling unit 15 will regularize the match- 
ing map and also provide a more efficient representation 
of images for further processing. Re-sampling unit 15 
10 receives input of the dense matching in three images 
from constraint match propagation unit 14 and outputs 
a list of re-sampled trinocular matches. 
[0097] The first image is initially subdivided into 
square patches by a regular grid of two different scales 
is 8 x 8 and 1 6 x 1 6. For each square patch, we obtain ail 
matched points of the square from the dense matching. 
A plane homography H is tentatively fitted to these 
matched points u, < — > u', of the square to look for po- 
tential planar patches. A homography in P 2 is a projec- 
tive transformation between projective planes, it is rep- 
resented by a homogeneous 3x3 non singular matrix 
such that ^u 4 , = Hu,, where u and u' are represented in 
homogeneous coordinates. Because a textured patch 
is rarely a perfect planar facet except for manufactured 
objects, the putative homography for a patch cannot be 
estimated by standard least squares estimators. Robust 
methods have to be adopted, which provide a reliable 
estimate of the homography even if some of the 
matched points of the square patch are not actually lying 
on the common plane on which the majority lies. If the 
consensus for the homography reaches 75%, the 
square patch is considered as planar. The delimitation 
of the corresponding planar patch in the second and the 
third image is defined by mapping the four comers of 
the square patch in the first image with the estimated 
homography H. Thus, a corresponding planar patches 
in three images is obtained. 

[0098] This process of fitting the square patch to a 
homography is first repeated for all square patches of 
the first image from the larger to the smaller scale, it 
turns out ail matched planar patches at the end. 

6 Three-view joint triangulation unit 

[0099] Now, athree-viewjointtriangulation unit 16 will 
be described below. The image interpolation relies ex- 
clusively on image content without any depth informa- 
tion and is sensitive to visibility changes and occlusions. 
The three view joint triangulation is designed essentially 
for handling the visibility issue. Three-view joint triangu- 
lation unit 1 6 receives input of the re-sampled trinocular 
matches and outputs joint three-view triangulation. The 
triangulation in each image will be Delaunay because of 
its minimal roughness properties. The Delaunay trian- 
gulation will be necessarily constrained as we want to 
separate the matched regions from the unmatched 
-ones. The boundaries of the connected components of 
the matched planar patches of the image must appear 



25 



30 



35 



40 



45 



50 



12 

SOOaO: <EP 1 1 34701 A2J_> 



23 



EP 1 134 701 A2 



24 



in all images, and therefore are the constraints for each 
Deiaunay triangulation. 

[0100] The joint three-view triangulation is defined as 
fulfilling the following conditions. 

5 

There is one-to-one vertex correspondence in three 
images. 

- The constraint edges are the boundary edge of the 
connected components of the matched regions in 
three images. 10 
There is one-to-one constraint edge correspond- 
ence in three images. 

In each image, the triangulation is a constraint De- 
iaunay triangulation by the constraint edges. 

15 

[0101] A natural choice to implement this joint three- 
view triangulation is a greedy-type algorithm. 



7 View interpolation unit 



20 



[0102] Now, a view interpolation unit 17 will be de- 
scribed below. According to view interpolation unit 17, 
any number of in-between new images can be generat- 
ed, for example, images seen from positions between a 
first and a second camera. These in-between images 25 
can be generated from the original three images. View 
interpolation unit 1 7 receives input of the three-view joint 
triangulation results and outputs any in-between image 
l(a, p, y) parameterized by a, p, and ysuch that a+0+y= 
1. 30 
[0103] The view interpolation processing is performed 
according to the following procedures. 

1 . The position of the resulting triangle is first inter- 
polated from three images. 35 

2. Each individual triangle is warped into the new 
position and a distortion weight is also assigned to 
the warped triangle. 

3. Each whole image is warped from its triangula- 
tion. In the absence of depth information, a warping <o 
order for each triangle is deduced from its maximum 
disparity to expect that any pixels that map to the 
same location in the generated image are arriving 

in back to front order as in the Patter's method. All 
unmatched triangles are assigned the smallest dis- 
parity so that they are always warped before any 
matched triangles. 

4. The final pixel color is obtained by bleeding three 
weighted warped images. 

so 

[0104] Furthermore, the similar idea developed for fa- 
cial image generation from 3 images could be extended 
to either 2 or N images with reasonable modification of 
the processing units. Other objects than face images 
could also be processed in a very similar manner. 55 
[0105] Needless to say, the present invention is not 
limited to the embodiment described above and may be 
varied within the scope of the invention described in the 



claims, and such variations are included within the 
scope of the present invention. 
[0106] As used herein, means is not limited to physi- 
cal means but includes cases where the functions of 
such means are realized through software. Further- 
more, the functions of one means may be realized 
through two or more physical means, and the functions 
of two or more means may be realized through one 
physical means. 



Claims 



A three-dimensional beauty simulation client-server 
system comprising: 

a shop-based client that obtains and transmits 
three-dimensional shape data regarding a us- 
er; and 

a server that comprises a makeup simulation 
unit that receives and stores said three-dimen- 
sional shape data from said shop-based client 
and carries out makeup simulation based on 
said three-dimensional shape data in response 
to the user's requests, and a data control unit 
that analyzes the user's operation record and 
generates administrative information. 

The three-dimensional beauty simulation client- 
server system according to claim 1 , further compris- 
ing a client that can access said server, wherein 
said server provides a makeup simulation in re- 
sponse to requests from said client. 

3. The three-dimensional beauty simulation client- 
server system according to claim 1 , further compris- 
ing a cellulartelephone that has a data transmission 
function and can access said server, wherein said 
server provides a makeup simulation in response 
to requests from said cellular telephone. 

4. The three-dimensional beauty simulation client- 
server system according to claim 1 , wherein said 
server further comprises a member registration unit 
that stores member registration information, and 
wherein said server provides makeup simulations 
to users registered beforehand in said member reg- 
istration unit. 

5. The three-dimensional beauty simulation client- 
server system according to claim 1 , wherein said 
server transmits the operation record and/or admin- 
istrative information regarding said users via a com- 
puter network. 

6. The three-dimensional beauty simulation client- 
server system according to daim 1, wherein said 
shop-based client comprises: 
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a plurality of cameras to obtain images of the 
user as seen from a plurality of viewpoints; 
a corresponding point search unit that receives 
each item of image data obtained from the plu- 
rality of cameras, analyzes the plurality of im- s 
ages, and searches for corresponding points 
that correspond to each other; 
a three-dimensional shape recognition unit that 
analyzes the searched corresponding points 
and recognizes the three-dimensional shape of 10 
the target object; 

a geometric calculation unit that sets a line of 
sight based on the recognition results from said 
three-dimensional shape recognition unit, and 
generates an image from a prescribed line of is 
sight through geometric conversion of the data 
based on the set line of sight; 
a display unit that displays the image generated 
by said geometric calculation unit; and 
communication means that transmits the image 20 
data generated by said geometric calculation 
unit to said server. 

7. The three-dimensional beauty simulation client- 
server system according to claim 6, wherein said 25 
corresponding point search unit and said geometric 
calculation unit comprises: 

a feature point extraction unit that extracts fea- 
ture points from each of said plurality of images; 30 
a correlation calculating unit that seeks con-e- 
lation among the feature points of said plurality 
of images and seeks combinations of said fea- 
ture points; 

a matching unit that discards combinations 35 
having a low feasibility from among said com- 9. 
binations of feature points based on the condi- 
tion that the images were seen from said plu- 
rality of viewpoints; 

a camera orientations determining unit that 40 
seeks the positions of said plurality of view- 
points and the directions of the lines of sight; 
and 

a match propagation unit that, under the condi- 
tions imposed by the positions of said plurality *s 
of viewpoints and the direction of said lines of 
sight obtained by said camera orientations de- 
termining unit, selects combinations of feature 
points starting with those having superior geo- 
metric and statistical reliability and adjusts the 50 
analysis range of the images of said target ob- 
ject. 

8. The three-dimensional beauty simulation client- 
server system according to claim 6, wherein said 55 
corresponding point search unit and said geometric 
calculation unit comprises: 



a feature point extraction unit that extracts fea- 
ture points from each of said plurality of images; 
a correlation calculating unit that seeks corre- 
lation among the feature points of said plurality 
of images and seeks combinations of said fea- 
ture points; 

a matching unit that discards combinations 
having a low feasibility from among said com- 
binations of feature points based on the condi- 
tion that the images were seen from said plu- 
rality of viewpoints; 

a camera orientations determining unit that 
seeks the positions of said plurality of view- 
points and the directions of the lines of sight; 
a match propagation unit that, under the condi- 
tions imposed by the positions of said plurality 
of viewpoints and the direction of said lines of 
sight obtained by said camera orientations de- 
termining unit, selects combinations of feature 
points starting with those having superior geo- 
metric and statistical reliability and adjusts the 
analysis range of the images of said target ob- 
ject; 

a resampling unit that normalizes the matching 
map obtained by said match propagation unit; 
a three-dimensional position measurement unit 
that determines the position of said target ob- 
ject in a three-dimensional space based on the 
normalized matching map; and 
a view interpolation unit that generates images 
seen from viewpoints different from said plural- 
ity of viewpoints based on the determined 
three-dimensional position of said target ob- 
ject. 

The three-dimensional beauty simulation client- 
server system according to claim 6, wherein said 
corresponding point search unit and said geometric 
calculation unit comprises: 

a feature point extraction unit that extracts fea- 
ture points from each of said plurality of images; 
a correlation calculating unit that seeks corre- 
lation among the feature points of said plurality 
of images and seeks combinations of said fea- 
ture points; 

a matching unit that discards combinations 
having a low feasibility from among said com- 
binations of feature points based on the condi- 
tion that the images were seen from said plu- 
rality of viewpoints; and 
a match propagation unit that, under the geo- 
metric constraints imposed by the lines of sight, 
selects combinations of feature points starting 
with those having superior geometric and sta- 
tistical reliability and adjusts the analysis range 
of the images of said target object. 
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10. The three-dimensional beauty simulation client- 
server system according to claim 6, wherein said 
corresponding point search unit and said geometric 
calculation unit comprises: 

5 

a feature point extraction unit that extracts fea- 
ture points from each of said plurality of images; 
a correlation calculating unit that seeks corre- 
lation among the feature points of said plurality 
of images and seeks combinations of said fea- io 
ture points; 

a matching unit that discards combinations 
having a low feasibility from among said com- 
binations of feature points based on the condi- 
tion that the images were seen from said plu- is 
rality of viewpoints; 

a match propagation unit that, under the geo- 
metric constraints imposed by the tines of sight, 
selects combinations of feature points starting 
with those having superior geometric and sta- 20 
tistical reliability and adjusts the analysis range 
of the images of said target object; 
a resampling unit that normalizes the matching 
map obtained by said match propagation unit; 
a three-dimensional position measurement unit 25 
that determines the position of said target ob- 
ject in a three-dimensional space based on the 
normalized matching map; and 
a view interpolation unit that generates images 
seen from viewpoints different from said plural- 30 
ity of viewpoints based on the determined 
three-dimensional position of said target ob- 
ject. 

1 1 . A three-dimensional beauty simulation server com- 35 
prising a makeup simulation unit that receives and 
stores three-dimensional shape data of a user from 
a shop-based client and carries out makeup simu- 
lation based on the three-dimensional shape data 
in response to requests from the user, and a data *o 
control unit that analyzes the operation record for 
said user and generates administrative information, 



with said requests, and a transmitting/receiving 
unit that transmits the output of said inf ormation 
processing unit to the requesting source and re- 
ceives requests from the requesting source. 

1 2. The three-dimensional beauty simulation server ac- 
cording to claim 11, wherein said makeup simula- 
tion providing unit analyzes the condition of the us- 
er's facial skin and the light and dark areas that in- 
dicate the protrusions and indentations thereon, 
and evaluates the user's facial expression based on 
the results of such analysis. 

13. The three-dimensional beauty simulation server ac- 
cording to claim 11, wherein said makeup simula- 
tion providing unit obtains a facial image of the user, 
displays for the user a plurality of target facial im- 
ages stored beforehand for allowing the user to se- 
lect one of these images, combines said user facial 
image and said target facial image in a plurality of 
predetermined ratios, and supplies a plurality of 
combined facial images to the user. 

1 4. The three-dimensional beauty simulation server ac- 
cording to claim 11, wherein said makeup simula- 
tion providing unit supplies facial images seen from 
freely chosen viewpoints. 

15. A system comprising means for deriving a 3-D im- 
age of a user, means storing information regarding 
user preferences regarding certain predetermined 
criteria, and means for generating an image of the 
user modified in accordance with said stored infor- 
mation. 

16. A method of producing a modified image of a user, 
the method comprising determining information re- 
lating to user preferences regarding certain prede- 
termined criteria, deriving a 3-D image of a user, 
and modifying the image of the user in accordance 
with said information. 

17. A method as claimed in claim 16 wherein the user's 
behaviour is monitored to determine said informa- 
tion relating to user preferences. 



wherein said makeup simulation unit comprises 
a receiving unit that receives said three-dimen- *5 
sional shape data, a database that stores the 
received three-dimensional shape data, and a 
makeup simulation providing unit that provides 
a makeup simulation in response to requests 
for such simulation; and so 
wherein said data control unit of said server 
comprises a user Information analyzer that re- 
ceives the operation history of the user and an- 
alyzes the trends therein, a control database 
that stores the analyzed data, an information ss 
processing unit that reads out data from the 
control database in response to external re- 
quests and processes the data in accordance 



«nonD:«EP 11 34701 A2 I > 



15 



EP 1 134 701 A2 



F 1G. 1 



MANUFACTURERS. 
MAGAZINE PUBLISHERS. 
ETC 



SERVER 



! MEMBER 
i IREG'STRATIQK 
UNIT 



; MAKEUP 

SIMULATION 
! UNIT 



OAT A 

CONTROL 
UNIT 



15 

<N 

CELLULAR TELEPHONE HAViNG 
OATA COMMUNICATION FUNCTION 
(INTERNET CELLULAR TELEPHONE) 



' 2 ~\J SHOP- 



; CLIENT 



BASED 




15a 

( 



— CAMERA 



OPERATEO TO PERFORM 
SIMULATION 



CLIENT 
(HOME-BASEO 
COMPUTER) 



SH0P-8ASED 
CLIENT 



T 



CPERATEO TO PERFORM 
SIMULATION 



OPERATEO TC PERFORM 
'SIMULATION 



123 



33 IMAGE 

PROCESSING 

APPARATUS 



USER 



123 



30 IMAGE 

CAPTURING 

APPARATUS 



ISOOCID: <EP 11 34701 A2_L> 



16 



EP 1 134 701 A2 



F IG. 2 



101 

Jl 



MEM3£R 06 



103 



30 FACE 
MODEL OB 



100 



\ 




MEM8ER 

REGISTRATION 
UNIT 










MAKEUP SIMULATION 
PROVIDING UNIT 



106 

V 



CONTROL OB 



105 

s I 



MEMBER REGISTRATION 
REQUEST 



SIMULATION SUPPLY 
REQUEST 

PROVIDE SIMULATION 



102 



30 FACE MODEL 
OATA RECEIVING 
UNIT 



3D FACE 
MODEL OATA 



USER INFORMATION 
ANALYZER 



INFORMATION 
PROCESSING UNIT 




«DOCIO:<£P 11 34701 A2_l_> 



17 



EP1 134 701 A2 




SDOCID: <EP 1134701A2 I > 



18 



EP1 134 701 A2 



F l G. 4 

4 



4b ( 4a 




- «i 



19 

iSnmO- <EP 1134701A2 I > 



EP 1 134 701 A2 



F I G. 5 



06TAIN TWO OR MORE IMAGES 
A. B. . . SEEN FROM TWO OR 
MORE DIFFERENT VIEWPOINTS 






I 




SEEK CORRESPONDENCE OF 
FEATURE POINTS AMONG 
IMAGES A. 6. ETC. 






i 




SEEK AMOUNT OF MOVEMENT OF 
CORRESPONDING FEATURE POINTS 
AMONG IMAGES A. 8, ETC. 


^ ,S3 




! 




SORTING 8ASE0 ON AMOUNT 
OF MOVEMENT 


^ M 


I 






IMAGE ORAW BEGINNING WITH 
FEATURE POINTS HAVING 
SMALLEST AMOUNT OF MOVEMENT 


^ ,S5 



20 

iSOOOO: <EP 1134701A2_I_> 



EP1 134 701 A2 



F IG 6 



1b 




ISDOQO: <EP 1 1 34701A2_I_> 



21 



EP1 134 701 A2 




100c 





ISOOOa <EP 1134701A2J_> 



22 



EP1 134 701 A2 




csnnr3T>*FP 1134701A2 l > 



23 



EP1 134 701 A2 



F I G. 9 

(2) 




SOOCJO: <EP 11 34701 A2_l_> 



24 



EP 1 134 701 A2 



F I G.10 



(a) 




SOOTH* <EP 11 34701 A2 I > 



25 



EP 1 134 701 A2 



F I G. 11 

it) 

i- — "1 

I 

i I 
l \ 




26 

SOOC1D: <£P 1134T01A2_I_> 



EP1 134 701 A2 



FIG. 12 



FIRST IMAGE 



103 
i / 



FEATURE POINT 
OETECTOR 



SECONO IMAGE 

10b 

!! <L 



THIRO IMAGE 



10c 
i / 



FEATURE POINT 
OETECTOR 



FEATURE POINT 
OETECTOR 



lia 



CORRELATION 
UNIT 



J ib 



CORRELATION 
UNIT 



ROBUST MATCHING UNIT 



SEED OISCOVERY UNIT 
~12 



AUTOMATIC CAMERA ORIENTATIONS 
DETERMINING UNIT 



MATCH PROPAGATION UNIT FOR 
PROPAGATING MATCHES FOUND 
UNOER GEOMETRIC LIMITATION 



J 



.13 



14 



RESAMPLING UNIT 



THREE-VIEW JOINT 
TR I ANGULATION UNIT 



VIEW INTERPOLATION UNIT 



.15 



16 



J7 



27 



EP 1 134 701 A2 



FIG. 13 



I £*S8U5,£f F,NE CAMERA TO 
I :u ??.OJECilQN CAMERA 



EXTR, 



CT EPIPCLSS | 



S12 



I fS £ ?¥ii. CA ^^ MATRIX 
1 AND CAMERA CENTER 



S13 



1 



I UPCATE PROJECTION STRUCTURE 



S14 



DETERMINE CAMERA 
C=IENTAT;ONS PARAMETERS 



S15 



iSOOClD: <£P 1 1 34701 A2_l_> 



26 



EP1 134 701 A2 



F I G. 14 



0ISCAR0 INITIAL SEED 


S21 


MATCHES HAVING BEST 




ZNCC SCORES 






i 




OISCQVER NEW MATCHES 
IN 'MATCH PROXIMITY* , 


S22 




AOO TO MATCH GROUP 





«nnnrv ,pp h.-utoiaj I > 



29 



THIS PAGE BLANK (uspto) 



EP1 134 701 A2 



F I G.15 




30 



1H» 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 
eTsKEWED/SLANTED IMAGES 

□'color OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: . 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



TH1SPMJEBLMUI**™ 



